DOCOHBttT BBSOHB 



PS 007 303 

Hone Start Evaluation Study. Interit Report III; 
Evaluation Plan 1973-1974. 

iibt Associates* inc. Caibridge, Mass. ; High/Scope 
Educational Research Foundation, Ypsilanti, Mich. 
Office of Child Development (DHEN) , Washington, 
D.C. 

DHBII-OS-72-127 
30 Aug 73 

46p.j For other reports in this study, see ED 069 439 
-ED 069 441, PS 007 273, PS 007 300 - PS 007 302, PS 
007 304 - PS 007 305 

MF-$0.75HC-$1. 85 PLUS POSTAGE 

♦Data Collection: ^Evaluation Technigues; Federal 
Prograss; *Hoie Visits; Objectivesj Parent 
Participation; ♦Preschool Prograis; ♦Program 
Evaluation; Program Improvement; Social Services; 
Testing 

♦Project Home Start 



This evaluation plan outlines the essential features 
of Home Start Evaluation activities scheduled for the twelve months 
beginning Fall 1973. Data will be collected in an effort to answer 
the following four guestions: (1) Are Home Start program guidelines 
being followed in the kinds of families and staff involved?, (2) Do 
families in Home Start for one year achieve greater progress toward 
program objectives than similar families not in Borne Start?, (3) Do 
Home Start children achieve the same developmental gains as Head 
Start children, and (4) What are the costs associated with Home Start 
and Head Start program operations? In addition to the guantitative 
data needed, qualitative data will be collected to describe the 
organization and activities of each of the sixteen Home Start 
projects. (Author/CS) 
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In tr oductio n 

This paper outlines essential features of Home Start 
Evaluation activities scheduled for the twelve months beginn* 
ing fall 1973^ This period begins the formal evaluation phase 
of the National Home Start Program, in contrast to the pilot 
phase which focused on refining the measurement methods while 
the sixteen projects were getting underway. During the coming 
year data will be collected to answer four questions of inter- 
est to policymakers in the Of fice of Child Development; 

• Arc Home Start program guidelines being followed in the 
kinds of families and staff involved in the progrartii and 
in the kinds of services provided? 

• Do families in Home Start for one year achieve greater 
progress toward program objectives than similar families 
not in Home Start? 

• Do Home Start children achieve the same developmental 
gains as Head Start children? 

• What are the costs associated with Home Start and Head 
Start program operations? 

In addition to the quantitative data needed for these four 
questions I qualitative data will be coll^^cted to describe the 
organization and activities of each of the sixteen Home Start 
projects. Similar information for the past year is included 
in the case studies which accompany this report. 

Data for the national evaluation are collected in three 
ways: firsts by an information system, which is a series of 
forms completed quarterly by local staff at each of the six- 
teen projects and sent to the evaluators for summarizing; 
second, by fall and spring site visits to each project by 
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evaluation staff; and thirds by a battery of questionnaires i 
rating scales # and tests administerecl to families fall and 
spring by local comunity interviewers* Results from the 
coming evaluation activities will be presented in three semi- 
annual reports • The overall time schedule for the coming 
year is summarized in Figure 1. Details of the evaluation 
activities are given in the next three sections; Information 
System, Project Site Visits, and Outcome Mes^bUires. 



Informa tion System 

The information system was evolved to its current format 
during the past year-and-a-half , and will continue essentially 
unchanged next year. The system is designed to gather basic 
statistics about family characteristics , staff characteristics, 
family referrals for services, and project expenditures. Three 
kinds of forms are used to obtain this information and summ- 
arize it: records , used by each project to obtain the raw 
information about'^individual families and staff? quarterly 
re ports , used by each project to summarize information across 
families and staff for submission to the evaluators; and 
nationa l reports , used by the evaluator to summarize inf orma- 
Hon aclross projects and time for submission to OCD policy- 
makers. A complete set of forms and instructions is presented 
in the Home S tart Information System Manua l (Abt Associates, 
1972) . ~ : ~ 

Every three months each of the sixteen Home Start pro- 
jects are scheduled to submit quarterly reports to staff in 
the National Home STart Office, who will them forward the 
reports to Abt Associates, Inc. Stiaff at Abt will tally the 
reports quarterly and submit semi-annual National Profile 
Reports with Interim Reports IV, V, and VI to the Office of 
Child Development. Copies of the Year Tally Reports and 
National Profile Reports will be sent to projects each quarter, 
accompanied by comments about reporting procedures. 

Abt staff will complete the following steps in compiling 
the National Profile Reports and Vear Tally Reports: 

• Quarterly reports submitted by projects will be checked 
for accuracy and completeness. If more information is 
needed Abt staff will call the projects. 

• A Year Tally Report will be updated quarterly for each 
individual project. 
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• National Profile Reports will be completed quarterly, 
summarizing totals for each Home Start project and 
the total for the National Home Start Program, 

• National Profile Report totals for each quarter will 
then be recorded on the Threo-Year National Profile 
Report. 



Pr oject Site Visits 

Overall site v isit plan . Unlike last year/ not all sites 
will be vis i tecri:he same number of times during the coming 
year. The six sites ^ participating in the summative evaluation 
will continue to be visited each fall and spring, but the other 
ten sites will be visited only during the spring. 

In four^ of the six summative sites Home Start evaluation 
staff will also visit local Head Start projects each fall and 
spring, to gather selected comparative information about costs 
and cost-related program characteristics. These four Head 
Start projects will also participate in the summative evalua- 
tion.. They are being included so that cost-effectiveness out- 
comes for Home Start can be judged against an objective yard- 
stick. Since the Head Start information is merely for overall 
comparison purposes, the information will not be used to formu- 
late recommendations for improving the Head Start projects, nor 
will Head Start and Home Start projects within individual sites 
be compared. 

In all, four kinds of information will be obtained by the 
evaluation staff as they visit projects; not all four kinds 
will be gathered from each project however, nor at each time 
point. 

• Pr oject updates and future plans ♦ Project staff will 
be asked, first, about organizational changes, such as 
changes in staff, in training procedures, in supervision 
methods, and in parent involvement; second, about pro- 
gress made toward achieving objectives set during the 
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previous site visit, especially those having the tnost 
success or those where problems have caused unusual 
delai^sr and third, about future objectives that project 
staff will try to accomplish by the time of the next 
site visit* An Interview form will be used to record 
information in a standard way across sites; 

• Home visitor/family interaction > Community interviewers 
will accompany Home Visitors on home visits to observe 
and record major characteristics of the visits, such as 
the person the home visitor works with most/ the acti- 
vities done, the materials VAsed, and the duration* 

Each Comnmnity Interviewer will visit two randomly 
selected families this way, for a total of eight to 
ten home visit observations per summative site. An 
observation checklist was pilot tested in spring 1973 
and will be refined for fall use. Related information 
about hiring, training, and supervising home visitors 
will be obtained by interviewing home visitors and 
their supervisors. 

• Selected progrm-wlde implementatl^^ features * Evalu- 
ation staff will gather In-depth information about 
aspects of program implementation that seem especially 
important for future use in planning and operating 
home-based service delivery programs. One or two aspects 
will be selected for Investigation during the spring 
1974 site visits, from among project characteristics 
such as the following: staff organization, staff train- 
ing, staff screening and hiring, methods for capturing 
community services, and parent roles in Home Start. 
Selection will be based on e valuator's judgments of the 
most important topics, in consultation with Washington 
Home Start staff. Checklists and interviews will be 
used to gather the necessary data. 

• Expenditure patterns and donated services . Project 
expenditures and donated services v;ill be described 
using a two-step process, by first recording predicted 
expenditures as described by the project directors, 

and Chen, on a later site visit, verifying the predicted 
figures using project financial records and staff inter- 
views. Predicted expenditures have already been com- 
pleted for Home Start projects, and will be obtained 
for the four Head Start projects in fall 1973. Verifi- 
cation will begin in the six summative Home Start pro- 
jects in fall 1973 and will continue in all Home Start 
and the four Head Start projects in spring 1974. A 
set of forms for securing this information was pilot 
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tested in spring 1973. Certain cost-related Informa^ 
tion about program characteristics, available from 
the information system for Home Start projects, wii.l 
have to be gathered for Head Start projects during 
the site visits. This includes number of staff, num- 
ber of children served, kind and level of major ser- 
vices, and some family characteristics. 

Figure 2 shows the overall relationships between projects, 
time points, and kinds of information. The data collection and 
analysis procedures for fall 1973 are presented next; specific 
procedures for spring and fall 1974 will be finalized later in 
the year. 



Site visit procedures for fall 1973 . Site visits to the 
six summative sites will take place between October 8 and 
October 19. One person will visit each site expressly to 
gather the site visit information, staying four days if 
assigned to the four Home Start/Head Start sites, and three 
days if assigned to the remaining two sites. Each site visitor 
will be assisted one day by the summative evaluation field moni- 
tor assigned to the site, who will be at the site to observe 
community interviewers administer the summative measures. The 
site visitors will spend one day in each site securing inf or-- 
mation for Home Start project updates and future plans, and 
the remaining time will be spent gathering Ciost Information 
for both the Home Start, and in four bites, the Head Start 
projects. The monitors will spend one day in each site ob- 
taining information about the hiring, training, and supervis- 
ing of home visitors* 

Pour experienced evaluation staff members will serve as 
site visitors. Two from Abt Associates will each visit two 
sites, and tv/o from the High/Scope Foundation will each visit 
one site. A two day training session is scheduled for late 
September at Abt Associates for the site visitors. During 
the same session the site monitors will participate in one 
day of training in addition to their training as monitors. 



Analysis of fall 1973 site visit information . Analysis 
of the fall 1973 site visit information will be completed for 
inclusion in the late February semi-annual Home Start report. 
Findings presented for Home Start projects will Include pro- 
ject updates and future plans, home visit descriptions, pro- 
ject expenditures, and donated community services. Findings 
presented for the Head Start projects will include test- 
related program characteristics, projected expenditures, and 
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donated services. Projected Head Start figures will not be 
verified until the spring 1974 visits, so only tentative 
comparisons will be i\^ade between Head Start and Home Start 
sites in the February 1974 report. 



Outcome Measures 

Experi mental design ^ Outcome measures will be adminis- 
tered to three separate groups of families, all having child- 
ren the right age to be eligible for two years of Home Start. 
An experimental group will consist of families entering Home 
Start in fall 1973. A control group will consis'c of families 
entering Home Start in Fall 1974, but identified for inclusion 
in the evaluation in fall 1973. Families from both groups 
v/ill be recruited at the same time and randomly assigned to 
one group or the other. A comparison group will consist of 
families with children enrolled in Head Start programs locat- 
ed in the same sites as Home Start programs. Families will 
not be randomly assigned to this group, but a subsample will 
be jcandomly selected from naturc^lly existing Head Start groups. 

The measurement battery v;ill be administered three diff- 
erent times, in October 1973, May 1974, and October 1974. 
The three measurement times will permit comparisons of Home 
Start, Head Start, and control families as they enter the 
program and after seven and twelve months of enrollment. The 
intermediate time was considered necessary to obtain families 
that might drop out of the program during summer vacation, 
and the final measurement time was selected to allow an ade^ 
quate (12 month) time for changes to occur resulting from 
program activities. Figure 3 shows the data matrix for the 
outcome evaluation design. 



FIGURE 3 
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Home Start 
entering spring 
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Sample . The battery of outcome measures will be adminis- • 
terod to families in six Home Start sites: Huntsville, Alabama? 
Dardanelle, Arkansas; Witchita, Kansas; Cldveland, Ohio; 
Houston, Texas; and Parkersburg, West Virginia. Decisions 
about sites to be included were based on judgements about their 
representativeness as well as on certain practical considera- 
tions. A nonrandom procedure was adopted at this stage because 
there were compelling reasons for not including certain sites, 
including site startup delays, cultural incompatibility of the 
measures, family migration, and geographic isolation. 

Forty families will be included in each of the three 
treatment groups in sites that have recruited a sufficient 
number of families . VJhere the total available for a group 
exceeds this number, forty will be randomly selected and the 
rest excluded from the evaluation . In most sites forty is 
the maximum number of the Home Start openings available. 
All forty are included in the fall 1973 measurement to ensure 
that a sufficient number will be available for measurement 
in the spring, after normal attrition occurs . No attempt 
will be made to replace families from any of the three groups 
leaving the evaluation during the year, since no entering 
measures will be available from replacement families to serve 
as a base for assessing change. 

Random assignment of families to the experimental and 
control groups, and random selection of families where more 
than forty are available for a group , will be performed by 
staff at the High/Scope Foundation using family rosters 
submitted by the Home Start and Head Start projects. Families 
will be stratified by home visitor before random selection to 
ensure a workable match between families and available open- 
ings in each project. Non-English speaking families and families 
with handicapped children will not be included in the data 
collection. Lists of families selected will be sent to the 
six sites shortly after they are received at High/Scope so 
program activities can begin as soon as possible. 

Mea surement battery . Nine measures will be in the 
battery during the coming year, including two children's 
tests, two child rating scales completed by adults, three 
parent questionnaires, a parent-child Interaction measure, 
and child height and weight. A list of the measures follows : 

• Preschool Inventory 

• Denver Developmental Screening Test 
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^Sohaeffer Behavior Invontory 



• Pupil Observation Checklist 

^ Illgh/Scope Home Envlronn^nt Scale 



• Parent Interview 

• Child Pood Intake Questionnaire 

• 8-13lock Sort Task 

• Height and Weight 

Pinal descriptions of the ineasures are presented in the 
current summative results volume accompanying this paper / and 
further information is available in Interim Reports I and II • 
Current plans call for giving the full battery to all three 
groups/ unless restrictions in project resources or site 
logistics necessitate a reduction in the number of measures 
administered to Head Start families. 

Data Collection , The battery of measures will be adminia- 
tered to families by community interviewers, ha.red locally in 
each site and trained in Michigan for six days before each fall 
and spring data collection begins • There will be five community 
interviewers in each site having a Head Start project , and four 
in each of the other two. In addition, each site will have a 
locally hired and specially trained site coordinator who assists 
in training, monitoring r and scheduling the community inter- 
vewers/ The site coordinators will help train the community 
interviewers in each six day training session, and in addition 
will receive a day of training specifically related to their 
scheduling and monitoring duties. 

Data collection will start in all six sites immediately 
following the training workshops each fall and spring. Family 
appointments and assignments of families to community inter- 
viewers for the first two weeks will be set up ahead of the 
workshop by the site coordinators. A site coordinator will 
accompany each community interviewer on her first administra- 
tion of each measure, to ensure that the proper procedures 
are followed. If necessary, additional training will be pro- 
vided to community interviewers before they are permitted to 
visit another family, and if that is not sufficient to achieve 
the necessary standards of administration the community inter- 
viewer will be dismissed. During the third and fourth week 
of data collection site monitors from High/Scope and Abt 
Associates will accompany each community interviewer on one 
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visit to determine whether field procedures are being followed. 
Site coordinators will continue to accompany conununity inter- 
viewers on visits each week that the monitors are not on site, 
visiting some Home Start families, some control families , and 
the Head Start centers in sites having them. No family is 
£icheduled to have more than one visit by either a monitor or 
site coordinator* 

Each community interviewer will administer the battery 
in two parts, during two separate visits to each fanily. 
Additional visits may be scheduled if families prefer, or if 
necessary to finish the battery. Testing will be done in 
the home for Home Start and control families, and in the Head 
Start centers for comparison children. Only famines who have 
signed permission letters will be included in the data collec- 
tion. Families will be paid five dollars each time they com- 
plete a full battery of measures. Families will be assigned 
to community interviewers randomly in urban sites, and by 
geographic region in rural sites to reduce costs. A special 
effort will be made to see that each community interviewer 
has an equal number of Home Start, control, and Head Start 
families. An effort will also be made to achieve an ethnic 
match between families and community interviewers. Each 
community interviewer will administer the battery to approxi- 
mately 18 to 22 families during the six week data collection 
period. 

Community interviewers will forward all data collected 
each week to the site coordinators . The site coordinators 
will log all the completed measures and check them for com- 
pleteness and obvious scoring errors. Following this review, 
the site coordinator will meet with community interviewers 
or call them as necessary to correct discrepancies. The site 
coordinators will then forward data to abt associates each 
week for a more thorough quality review before they are sent 
to the High/Scope Foundation for coding and analysis. 

A more detailed plan of the field data collection pro- 
cedures is presented in Appendix A to this paper. 

D ata reduction . High/Scope Foundation staff will convert 
the data to machine readable form after they are received from 
Abt Associates. A series of fixed steps will be followed: 

• A log of all incoming fom^s will be maintained. 

• Site, family, and child identification numbers will be 
assigned to each form. 



ERLC 



• Pormats f oj: coding each item from each test> rating 
scalo/ or questionnaire will be developed and recorded 
in a coding manual # 

^Reaponses for each item will be entered into computer 
■ disk ■ files 

• The files will be screened for errors two ways: 

first/ by directly comparing the computer file printouts 
to the original protocols , and second , by us ing computer 
programs to find invalid codes # unreasonable values, 
and missing data « 

• The verified files will then be used to build working 
files compatible with available statistical analysis 
programs* 

• Original protocals will be recorded on microfilm for 
permanent storage* 

Da ta analysis < Three kinds of statistical analyses will 
be performed/ Including description of characteristics of the 
measurement battery, comparison of pre- to post changes in the 
three groups, and identification of interrelationships among 
variables. 

Description of the measurement battery characteristics 
will be carried put through analyses of the internal character- 
istics of measures and analysis of whole scores* The following 
internal characteristics will be looked at for each measure s 

• Response distributions for each item; 

• Percent of persons passing each item; 

• Intercorrelations among items; 

j» ■ 

• Factor structure among items; 

• Internal consistency reliability* 

Characteristics of total scores and subscores that will 
be examined include: 

• Means 

• Standard deviations 

• Standard errors of the means 
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♦ Intercor relations with other scores 

• Factor structure of the entire battery 

All of these characteristics were examined for the spring 
1973 measurement battery and reported In the current^ 
results volume. The purpose for repeating these analyses next 
year Is to determine If the early findings are stable/ provid- 
ing an estimate of how much confidence can be put in the hattery. 

Comparisons of change over time for the three treatment 
groups will be made in two ways: by using analysis of variance 
methods to assess the average change # and by using nonpara-* 
metric item change methods to determine the nature of changes 
both within and across individuals • The most direct teet for 
change using analysis of variance is a tedt of the first or<?or 
interactions between the Home Start, Head Start and control 
groups* Since two sites have older children in the program 
than the other sites I child age will be used as a covariate 
or blocking factor in the analysis of variance » Estimates of 
the magnitudes of effects using multiple squared correl^ition 
coefficients will be obtained/ as well as calculations of the 
power of the tests for the purpose oC determining Type II 
error rates. Post hoc comparisons will be performed as needed 
to interpret the ANOVA results. 

Interrelationships between scores will be examined using 
Gorrelations, factor analysis, and multiple regression/ These 
analyses are not necessary to answer the central valuation 
questions, but will be performed because of their possible 
Importance for peripheral questions • Examples of such analyses 
may be found in the summative evaluation results volume of 
the report. 

Re ports . Summative results will be presented in each of 
the three semi-annual reports. The first/ scheduled for late 
February, 1974, will describe the measurement battery character- 
istics and the entering population characteristics. Since the 
battery will be essentially fixed at that time, the focus will 
not be on recommending alterations to the battery, but simply 
replicating results presented in the summative results volume 
of this report. 

The second report, due at the end of August 1974 will 
present a comparison of the three groups over the first seven 
months of enrollment in Home Start and Head Start. Limited 
analyses of the measurement battery will be presented, but 
this will not be a major focus as in previous reports. Also 
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Included will be an analysis of the charactejriatics of fall 
scores for children who dropped out. 

Comparisons of the three groups over the full twelve 
month period will be presented in the fall 1975 report, along 
with overall conclusions about the outcome. of the twelve month 
formal evaluation. 
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EVALUATtON PLAN FOR FALL <73 TO FALL ^74 
Data Collection ^ Suxnmatlve Evaluation 

The field operations described in this evaluation plan to collect data for the 
suKuttative evaluation during the Pall of 1973 will remain essentially the same 
for the remainder of the evaluation since no change in the scope of work r the 
number of families to be tested, and other parameters are anticipated, 

C oirpbnents of the Data Collection Effort 

The following flow chart indicates the three components of the summative evalua- 
tion data collection effort for the Fall (recruiting, training, and testing) and 
specific tasks which need to be completed. 
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Fall of 1973 - Staffing 



The following field staff organization is proposed for the Fall 
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The expansion of the size of the field staff as noted on the previous page is 
necessary for the following reasons: 

^ to allow for attrition between the time of the training conference and 
the start of testing2 and to be able to substitute for testers who drop 
out of the program during the course of field operations;^ 

• to insure tinely completion of testing {especially during the Fall Nihen 
heavy rainfalls in some of the sites could interfere with field opera- 
tions) ; 

# to make the number of testing visits per community interviewer manage- 
able for part-time field staff. 



^ The nvimber of vacancies in the field staff for the Fall of ig73 are indicated 
on the charts 

12.5% attrition during the Spring of 1973. 
^ Another 4% of the field staff dropped out during the course of testing. 



Recruitment of Field Staff 

Procedures used for Spring recruiting of the field staff will again be used for 
the Fall. In summary/ the procedures cal,;|. fors 

• advertising positions in local news|>aiier'&/ posting the jobs with the 
local employment agency; and asking Home Start Program staff to sug- 
gest and identify candidates for the position; i < 

• on-site interviewing of candidates interested in the position by staff 
from Abt Associates Inc. and/or the High/Scope Education's! Research 
Foundation; 

• giving the Home Start Program Director/Coordinator an opportunity to 

screen applicants and review the pool of qualified candidates if she 
so desires; 

• give consideration to the geographic distribution of the families 
selected for testing and to hire field staff in those specific com- 
munities to reduce travel time and cost. 

During the Fall, site coordinators will assist in the recruitment process. 
They have been asked to identify candidates for the job and to set up an appoint- 
ment schedule with applicants to be interviewed by Abt Associates or High/Scope 
staff. Site coordinators will be provided with a supply of fact sheets to be 
given to applicants, providing some general information on the Home Start Pro- 
gram and the evaluation, and specifics regarding the job of community inter- 
viewer. 
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site Start-Up and Preparation 

Somo start-up activities will be conduoted prior to tho start of training so that 
testing! can start iromediately following the training conference. Prompt start-up 
of testing is necessary to insure that all testing is completed prior to November 
15* Start-up activities prior to the training conference will consist of: 

# a letter from Abt Associates and High/Scope to Directors/Coordinators ^ 
of the Home and Head Start Programs {where applicable) to outline the 
Fall testing schedule and basic field procedures to be followed by tho 
field staff; 

•mailing of preliminary testing rosters for families enrolled In the 
Home Start Program who were selected for the Fall testing j 

# telephone conferences with the site coordinators regarding initial 
assignments of families to community interviewers and scheduling 
requirements for the first two weeks following trainingi 

■ ■ * ^ 

# site coordinator meeting with the Director/Coordinator of the Home 
Start Program to discuss schMuling pr^ . 
schedule (in some sites this may simply be a telephone conversation 
with the Director); 

0 site coordinator meeting or telephone conference with Home Visitors 
to set up testing schedule for the two weeks following training; 

•preparation and mailing of testing schedules to Home Visitors and 
sending families a notice reminding them of the date and time of the 
testing ivisit. 

If time allows,: a mee^ting will also be scheduled by tho site coordinators with 
the Director/Coordinator of the Head Start Program (where applicable) to dis- ^ 
cuss the Pall testing schedule and field procedures. 

In Alabama/ the new site coordinator will be given considerable assistance in 
scheduling by the outgoing coordinator and central control at Abt Associates Inc» 



During the first two weeks of testing, visits will only be made to families 
enrolled in the Hoiae Start Program. Complete testing rosters will be made 
available to the site coordinators at the time of the training conference. 
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Training 

Three types of training will be conducted during the Pall t 

• sito coordinator i ning in field procedures and logistics - one day? 

# training of trainers (including the site coordinators who will be in- 
volved in the training as co-trainers); in addition to reviewing in- 
depth the revised test battery and protocols, considerable tirae will 
be spent with the training staff on monitoring and start-up procedures 
- three days; and ; 

% community interviewer training in the test battery - six days* 

A training schedule similar to the one used for the Spring will be followedi with 
only slight modifications. During the Pall, community interviewers will be 
trained in the revised test battery, in addition to the Home Visit Observation 
Instrument which will be administered to a liirdted sample of families. Two home 
visits will again be scheduled for practice sessions during the week of training. 

Protocols will again be used as the principal training device for the three child 
measures. Revision of the PSI and DDST protocols will start when approval of the 
test revisions have been obtained by the Office of Child Development. Protocols 
also will be developed for the Concept Development Test. In addition to the pro- 
tocols, quizzes which were used during the Pall of 1972 will be revised and used 
as another training mechanism. 

Under an <-greement witli the Home Start Program in Gloucester, video tapes of home 
visits are presently being prepared to be used at the training session. Community 
interviewers will be trained in scoring and observation techniques using these 
video tapes t 

The tollowing chart indicates how the training conference will be staffed during • 
the Fall: 
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Both the test specialist and training coordinator will assist the snaall group 
practice sessions throughout the week, especially the Alabama group since the 
site coordinator will not be in a position to act as a co-trainer. 
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Monitoring 

immediately following the training conferenc^^ testing will start in all six 
sites* Testing schedules wiir^iJ^'oady been set up for the first two weeks 
of testing by the site coordinator and will need to be confirmed following the 
training conference. Site coordinators Will have major responsibility for statt- ' 
up operations and monitoring.^ During the first week of testing, each community 
interviewer (including those experienced in testing) will be accompanied on a 
testing visit at least once * The site coordinator and community interviewer will 
meet following the visit to discuss test performance and compare test scores. If 
the community interviewer ^s performance is satisfactory, she will be given appro*- 
val to continue testing throughout the remainder of the week. If her performance 
is not satisfactory^ the site coordinator will accompany the community :^nterviewer 
on a second testing visits following a review session on the test battery/ If 
the community interviewer's performance continues to be unsatisfactory # a deci- 
sion will be made by Central Control at Abt Associates Inc» regarding the con- 
tinued, employment of that tester. This will be done following the second testing 
visit* In some cases it may be advisable to have the site coordinator conduct 
additional review sessions and monitor a 3rd and 4th visit. 

During the first week of monitoringi community interviewers only will be acbninister-* 
ing the test battexry ^for the first visit. During the second week r community inter-* 
viewers will be monitored on their performance un the second visit battery. During 
the second week, procedures for monitoring which are outlined above will be followed. 

Attached is a proposed schedule fojc site coordinator monitoring for the first and 
second weeks of testing. 



In Alabama / a different procedure will be used for start-up and monitoring. 
The site monitor responsible for that site will conduct the monitoring, with 
the new site coordinator accompanying her on at least half the visits* 

I^erfonnance standards will be developed by Abt Associates Ino* prior to the Pa31 
field effort. 
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BEST COPY AVAIWBLE 

the aite monitors from lligh/Scopo and Abt Associates (persons responsible for 
training conw\unity site visits during the 3rd and 4th 

week of testing^ The monitor *s role on site will only be to spot check accur- 
apy of t^st administration r test score reliability, and to detennine whether 
field pirocedures are boing followed. It is proposed that the site monitor 
accompany each community interviewer on at least one visit (some first and 
second visits) Following the testing visits, the site monitor may schedule 
a ineeting with the entire field staff to review problem areas. Site coordlna^ 
tors will not accompany the monitor on these testing visltSf except in Alabama 
as was noted previously. Monitors, in addition to monitoring test performancei 
will also have responsibility for collecting data for the formative evaluation. 

On-going monitoring will be provided by the site coordinators t During week 3 or 
4# depending on the date of the site monitor's visit (from High/Scope and Abt 
Associates} I the site coordinator will accompany each community Interviewer on a 
testing visit to a Head Start Center (if applicable). If no Head Start testing 
is Involved on site, these testing visits will be .to control group families. 

The chart below indicates the number of monitoring visits that will be made during 
the Fall. It should be noted that no monitoring visits are planned to observe the 
small sample of home visits. The presence of a monitor in the home might affect 
the conduct of home visit activities planned. 

An ^ihhftmpt will be made during the Fall to limit the number of monitoring visits 
to a particular family to one . 
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HiaiBBOfflThese Visits will be to control group families. 



Field Operations 

Tho following ptocedutrcs are proposed for the remainder of the evaluation: 

(1) Po rtnission Siips 

Home Start Program staff will again be requested to obtain signed letters 
d£ permission from families selected for enrollment in the program and 
from those who will be part of the control group. It is reconunended that 
parents sign these letters of permission as part of the intake/enrollment 
procedure. Letters of permission will be forwarded to Abt Associates Inc. 
during the first two weeks in Septeiftber^ prior to the start of testing. 

'A slightly different procedure for obtaining permission slips will be 
followed for Head Start families selected for tho testing • As soon as 
the sample has been selected and the rosters hav^ been sent to the pro-* 
graxtvs^ Head Start Directors/Coordinators will be requested to obtain 
letters of permission from the parents. Rather than having the programs 
send the letters to Abt Associates Inc.i site coordinators will be res- 
ponsible for insuring that permissions have been received from the parents 
prior to the start of testing. Site coordinators or comunity Interviewers 
will then forward the letter of pertnissioa with the testing data for that 
particular family. 

(2) Family Assignments 

Initial family assignments will have been made prior to the training con- 
ference to enable site coordinators to schedule testing visits for the. 
two weeks following the training. The remainder of the assignments will 
be made at the training conference after the sample lists for the treat- 
ment group, the control group, and Head Start have been reviewed by the 
coordinator of field operations and the site coordinators. While a ran- 
dom assignment' procedure will be used for urban sites, families in rural 
areas will b^ assigned by geographic region to reduce travel time and cost/ 
For example, a cdirxnunity interviewer residing in Wood County will be res-* 
ponsible for testing all families in that county and other surrounding 
areas. 

V 
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*tho assignments wade at the training conferonce are tentative and will be 
reviewed by the Home Start Director/Coordinator to insure that the ethnic 
match of community interviewer-family is acceptable to the program.^ Pol- 
lowing this meeting with the Director I a family assignment sheet Vlll be 
prepared for each community interviewer , copies of which will be sent to 
Home Start, or Head start, and Central Control at Abt Associates Inc, Site 
coordinators will retain a copy of the assignment sheets. On the listS/ 
information will be provided as to whether the family is enrolled in Home 
Start or Head Start; or whether the family is part of the control group* 
Community interviewers need to have this Information so that proper field 
procedures can be followed (see subsequent sections) « 

Although Home Start Programs are required to recruit 40 new families for 
enrollment in the Fall, and 40 families for delayed entry into the pro- 
gram, it is anticipated that only approximately 35 families in each group 
will be available for testing. This reduction in the sample size will pri- 
marily be the result of family turnover in the six summative evaluation 

sites, unless programs have over-recruited for both the treatment and con- 

2 ■ * ■ ^ ■ ■ ■ 

trol groups. 

The following charts indicate the number and distribution of families 
assigned to each coKimunity interviewer in both Head Start and Non-Head 
Start test sitess ^ 
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During the Spring, two programs made changes in family assignments since they 
objected to having black community interviewers test white families. 

19?i of the families selected for testing in the Spring did not participate in 
the summative evaluation due to termination from the program, illness or other 
emergencies, and uncooperation on the part of the family. 
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(3) Scheduling of Tosbing Visits 

Site coordinators will be responsible for scheduling testing visits to 
faroilies in the treatment, control ^ and Head Start Group, Specific proce- 
dures are outlined below for scheduling visits for each of the three test 
groups. Before scheduling, can start, site coordinators need to obtain and 
review time availability sheets from each community interviewer. On these 
sheets, field staff indicate commitments they have during the period of 
testing and other times which are not convenient for testing. 

Following are sched|iling procedures for each of the three test groups t 

(a) Treatment Group 

While Home Visitors were requested to accompany the community ■ 
interviewers on testing visits during the Spring, the Home Start 
Programs will be given the option of having community interviewers 
conduct the testing on their own, in order to minimize interrup- 
tions in regular program activities. If Home Visitors elect not 
to accompany community interviewers on testing visits, procedures 
as outlined in (b) will be followed* 

If Home Visitors plan to be present during the testing visits to 
treatment families, procedures will be as follows for scheduling: 

m the site coordinator will meet or talk with the Home Visitor on 
the telephone to set up a testing schedule which is convenient 
for both the community interviewer, the Home Visitors, and the 
families. Unfortunately, it is not always possible to conduct 
the testing visit at the regular Home Visit time because of con- 
flicts in field logistics* 

• following this meeting or conversation by telephone, the site 
coordinator will prepare a testing schedule for the Home Visitor, 
a copy of which will be forwarded to the Program Director. 

# a postcard confirming the date and time of the testing visit 
will be sent to parents. Parents will be asked to contact the 
site coordinator or community interviewer if the appointment 
needs to be changed. Confirmation cards will be mailed five 
days prior to the visit. 
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Procedures for rescheduling testing visits to the treatment group 
have been slightly modified since the Spring, Community inter- 
- viewers will be permitted to tnake changes to the testing schedule 

directly with the Home Visitor (or the parent if the Home Visitor 
is not present) r provided that the chcuige does not conflict with 
other scheduled testing visits and that the site coordinator is 
notified of the change. On a weekly basiS/ sit^ coordinators are 
responsible for preparing an xipdated testing schedule for Home ; 
Visitors and community interviewers. 

A third visit needs to be scheduled for 0-10 families in tho treat- 
ment group so that each community interviewer can observe two homo 
visits (8 for non-Head Start and 10 for Head-Start sites)/ Families, 
will be randomly selected by High/Scope for the home visit observa- 
tions. Schedules will be set up by the site coordinators directly 
with the Home Visitor^. Since these visits will be at the regular 
home visit tiiue, there is no need to confirm the time and date of 
this visit with the family. This third visit, however, must be 
Included on the schedules of testing for the Home Visitors and the 
community interviewers. 

(b) Control Group 

• ^ letter will be 6ent to parents in the control group (and 
families in the treatment group who will be visited by field 
staff without the Home Visitor present) approximately two 
weeks after the programs have been notified of the selections. 
The purpose of this form letter is to provide some general 
information regarding the evaluation and the testing and to 
advise the parent that the site coordinator will be in»touch 
with her in the near future to set up an appointment for test- 
ing. 

# a telephone call to families if a telephone is present in the 
home to arrange the first testing visit at their convenience .^:^^v 
The site coordinator v;ill insure th^t the visit is scheduled. 

at a time when older siblings to be tested can be present. 

# this telephone conversation will be followed up by a p ost- 
card confirming the appointment. 
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# for families who do not have a telephone, site coordinators 
will sot up preliminary testing schedules- The SC will then 
inform the parent of the date and time the visit has been 
scheduled, asking the parent to get in touch with the site 
coordinator if it isn*t convenient* In rural sites, a pro- 
cedure v/ill be set up whereby parents can call site coordi- 
nators collect if the call is long distance. 

Problem areas — Our experience indicates that parents frequeAtly 
do not keep appointments even if they have been reminded of them 
several times. Parents also may not be responsive to correspon- 
dence regarding the testing visits from the site coordinator. It 
is suggested that no more than three attempts be made by the com- 
munity interviewer to reach the family* V 

# if the first visit is completed, community interviewers will 
set up another appointment with the parent for tho second 
visit. A postcard confirming the date and time of this visit 
will be sent to the parent five days prior to the visit. Com- 
munity interviewers will be responsible for notifying the 
site coordinator of the date of the second visit* 

# if a third visit is necessary, a similar procedure will be 
followed as is outlined above. 

Head Start Group 

# site coordinators will arrange a meetin g with Head Start Pro- 
gram Director and local program staffs (if applicable). The 
purpose of this meetinv} is to discusjb acheuuliug and testing 

' procedures • 

# site coordinators will then arrange to personally visit the 
local Head Start programs to set up schedules with local staff 
or call the programs. Blocks of time will be set aside for 
testing Head Start children — for example 3 mornings each 
week if this is convenient for the programs. 

e follo*"?ing the meeting or telephone conversatJon, site coordi- 
nators will pjtepare a testing schedule for Head Start staff 
to be updated' weekly. 

# five days prior to the testing visits parents of the Head 
Start children will be notifie^^ of the date and time of the 
visit. Parents may elect to be present while testing takes 
place. Depending on which tests are selected for the Head 
Start battery, parents may be required to be present for the 
8-Block Sort Task. 



Problem areas — Testing visits may be difficult to schedul^^ 
if parents need to be present, since many of them may work 
and may not be available during day-time hours. Special 
evening visits, away f rota the Head Start Center, may need 
to be arranged for these families. 

Tes^^^ng 

■ ■ T . '. ■ ■ . ■ ■ 

Testing procedures used foi: the Fall will remain virtually unchanged. An 
attempt will be made to complete the entire test battery In two visits, 
rather than three • it will be stressed to the parents, however, that they 
may terminate a visit if they feel the child is getting too tired or the 
mother has other commitments. Some procedures need to be developed for the 
the community interviewers to assist them in testing situations where sib- 
lings are present or no other adults are in the home to play with the focal 
child when the 8-Block Sort Task is being taught to the mother. 

Head Start children will be tested in the center, unless 8-Block require^ 
ments dictate otherwise. An a tt^pt will be made to conduct testing ses-^ 
sions outside of the regular classroom setting, but in a room with which 
the child is familiar, to avoid distractions or interference by other chil^ 
dren. if testing is conducted in the elassro^ quiet corner of the room 
should be found. Presence of Head Start staff during testing will be limited 
to one person. 

Parent Cards 

Community interviewers will be res^jonsible during the Fall to obtain the 
signature of the parents on a parent card indicating the two (or in the 
case of Head Start one) visit (s) have been completed. Signed parent 
cards will be included with the testing materials for that particular 
visit • If Head Start parents are not present for the testing, local Head 
Start staff may sign for the parent. 



Two procedures for paying parents $5.00 for thoir particlpAtion in the 
evaluation have been set up for .the treatment group; 

(1) Home Start Programs weekly will be paid $5.00 for each com- 
pletion of a test battery to be. distributed to the parents 
(Programs must indicate prior to testing, howeveti which 
procedure they will be using). 

(2) Upon receipt of the parent cards, a thank-you note and a 
check for $5.00 will be sent to the parents. . 

Both for the control and Head Start Groups the second procedure will be 
used, unless Head Start prefers the other payment procedure* 

(6) Data Check-In 

Community interviewers on a weekly basis will forward all testing materials 
to the site coordinator for data check-in. The cite coordinator will log 
al^ test materials and check the tests and questionnaires for completeness 
and obvious scoring errors* Following this review, the site coordinator 
viir meet with the community interviewer or talk with her over the tele- 
phone to discuss discrepancies. Data is then forwarded to Abt Associates 
Inc. for a more thorough review before it is sent to High/Scope for coding 
and analysis « 

< 7) Forms to be Used in the Field Effort 

The following records will be kept by field staff during the Fall evaluation 
effort: 

# Time Availability Sheets for each community interviewer indicat*- 
ing availability for testing; 

# Family Assignment Sheets for each community interviewer; 

♦ Testing Scheduled for Home Start and Head Start staff and 
ccanmunity interviewers; 

♦ Time and Travel Logs which are submitted weekly to Central 
Control at Abt Associates inc. On the logs, community in- 
t'>rvlewer8 and site coordinators make entries for each 
visit they make regarding the time spent in the ho/c\e and 
mileage to and from the home; 

^ Log of Unsuccessful Visits > On this log, community inter- 
viewers indicate the number of visits that wet'e cancelled 
without notification* Logs are submitted to Central Con- 
trol weekly; 



♦ Hallin<i Logs. Weekly, coiwnunity interviewers indicate on 
this log the testing materials that were sent to the site 
coordinator. Upon receipt of this log and after checking 
the materials, a flgned copy of the log will be returned to 
the community Interviewer to indicate that the materials 
have been received^ 

• Communications Logs on which si t-^ coordinators record ail 
telephone calls with fa.v.Uies, program staffs and community 
interviewers. On this log, site coordinators also note 
action that needs to be or wa-j taken > and 

.• Wonltoring Reports which ara ^nmpTot-ori Ky ^^^.^^^p^^^,. 

or coordinator following each monitoring visit noting errors 
in test administration. 

A detailed field procedures manual will be prepared for use by community inter- 
viewers and site coordinators. 



Central Office Control 

The primary function of central office control is to coordinate all testing 
activities among the six summative evaluation sites, Responsibilities of cen- 
tral office control include the following; 

• liaison between field staff and Home Start and Head Start Programs; 

3 •supervision of site coordinators and other field staff , with direct 
responsibility for making decisions regarding the continued employ-- 
ment of field staff; 

• coordination of monitoring effort and activities of site monitors; 

• insuring that field procedures are beiiig followed on site and timely 
completion of testing; 

# provide technical assistance to field staff on administration and 
scoring procedures and other field problems; 

• review and logging of all testing materials; 

# review and analysis of all monitoring reports; and 

^# review of time and travel logs from field staff and payment- 

During the Fall field effort, central control will compile the following 
reports I 

0 Statistics and analysis of scoring errors both from the training 
conference and score forms 6f site monitors and coordinators; 

# Statistics and analysis of administration errors from the monitor- 
ing reports; * 

# h rep6tt on the number of unsuccessful viijits and the ft of test 
c batteries that were administered in mbre than two visits; and 

; # A report on completeness of data, attrition of field staff, and 
iti^jor problem areas* 
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FORMATIVE EVALUATION PLAN 



Introduction * 

The following illustration* is presented to give an overview of the size 
^nd shape of the formative portion of the evaluation effort. As shown 
below a roajor modification of the evaluation initially designed involves 
the on-going collecting of cost information (both federal expenditures and 
in-kind services) at four Head Start programs in locales served by four of 
the six summative Home Start programs. For this reason, the plan for forma- 
tive evaluation of the ten remaining Home Start programs is presented after 
the six Home Start summative sites and four Head Start comparison programs. 
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■Vr»^!r>- * = ^Clitrcles refer t6 times data will be collected. 



Pro c edures 

Along with the Fall-Spring test cycle a Fall-Spring set of site visits will 
be made. Site visits concentrate on collecting data relative to the directic 
and effeciency of program implementation. The collection^ analysis^ and 
reporting of this implementation dat^ serves at least four purposes. 

First, this process is formative, that is it identifies recurring areas of 
relative strength and weakness regarding program implementation* Following 
such identification, rec<xnmendations can be made to OCD regarding possible 
improvement of both program direction and efficiency. 

Second, in a more classical research sense, the process of documenting 
de facto program Implementation is also the process in which the actual pro-^ 
qriiin treatment is documented > As national demonstration projects cannotr- 
and do not intend— to maintain total conttol ovor the treatment, such docu^ 
mentation of actual implementation or treatment is necessary if the summative 
test findings are to be analyzed in a systematic manner. (Many research 
studies relate test findings to various demographic characteristics of the 
service population, e.g. the Ohio-Westlnghous^ Study. While such analysed 
may be helpful, they do not lead to identification of specific implementation 
variables that may be responsible for the significant changes that occur.) 

Third, if a systematic approach is used for collection and analysis of imple- 
mentation data, comparative studies can be made of different programs and 
treatments. (In this case Home Start and H6ad Start — see below*) 

Fourth I documentation and analysis of program implementation can produce an 
on-going history of recurring changes in the national program. As recurring 
areas of weakness or concern are addressed, the formative evaluation process 
can hopefully document program improvement strategies that have been success- 
fully employed at the national level. Such strategies may be helpful - ; 
variety of jpirogram intent on improving the quality <>t P^b^tam 



implementation. (It should be noted that as the formative process described 
here centers on the systematic documentation and analysis of recurring Imple* 
mentation issues, it follows that a pattern of catagorization and quantifi- 
cation must be employed. As a result key implementation issues will be treated 
in a somewhat generalized manner. Recommendations based upon these generaliza-' 
tions will also be broad enough to be applicable to at least several local 
programs which encounter similar implementation problems. In other words, a 
systematic approach to formative evaluation assumes that various bits and 
pieces of information regarding local program implementation will not be dis^ 
played (except in tabulated form) in the major program analysis report. Such 
site specific information will be preserved in the individual case study narra- 
tives and in the information system. (See the sections below on case study 
narratives and the information system. ) 

Site Visit Plan Fall 1973 and its Relation to the Previous Spring Visits 

In Spring of 1973 four-day site visits were made to each of the 16 Home 
Start programs. Information of three kinds was collected; Program Up-Date 
and Future Plans, Program Costs, and Home Visitor/Family Interaction. Most 
of this information was collected via instrximentaled interviews with the local 
staff. Certain project records regarding budgets, expenditures (especially 
salaries), and staff time utiliaation were also utilized v Finally, four home 
visits at each site were observed* and documented with the Home Visitor/Family 
Interaction— Observation Guide. 

In the Fall of 1973 site visits will be made to the six Home Start sites at 
which summative testing is being conducted. Site visits will also be made 
to the four Head Start programs which exist in the same locale as a Home 
Start Summative Testing Site. Sufficient data will be collected from the ^ 
four Head Start programs to enable the making of across site ( but not within 
site) comparisons of Head Start/Home Start expenditure patterns and comparisons 
of the degree and manner in which these Head Start and Home Start programs 
have utilized community services on an in-kind basis. Notei the Head Start 



visitations do not seek to document overall program implementation. The 
National Home Start Evaluation will not document the Head Start "treatment,'' 
that is it will not document or in any way evaluate the various programmatic 
efforts within Head Start, *In similar fashion, the Head Start site data will 
not be analyzed for the purpose of making recommendations for the improvement 
of any Head Start process. 

An^overview of the site visit plan for Fall 1973 is presented below. Tenta- 
tive Spring 74 plans will be discussed later in this section. 

Site Visit Schedule 

The Fall visits to the six Home Start summative sites, and to the Head Start 

programs in four of these six locales, will be staffed as follows. A staff 

person frcw\ either Abt Associates (four lodales) or High/Scope (2 locales) 

will spend four days at the combination Home Start/Head Start locales* 

(Alabama, Arkansas, Texas— Houston, and West Virginia). In each case this 
staff person will be supported by a testing monitor who will bo on site for 

2b days during the inclusive dates of October 8-19. The monitors will con- 
tribute one day at each site to supporting the site visit effort. (Although 
the support contributed by this one day of the monitor may be limited, the 
added perspective gained by a two person observation team han been found to 
be valuable. Certain logistical questionsi can also be addre&sed with the flex 
ibility provided by the two-person effort.) ' 

At the four Home Start/Head Start locales the major site visitor can devote 
one day to securing Home Start program implementation data and three days 
securing both Home Start and Head Start costs information (both on expend!** : 
ture of grant dollars and on the securing of in-kind community services. ) 
The monitor day can best be utili2ed In collecting programatic information 
on the Home Visitor process, especially its educational aspects. (The monitors 
are either consultatnts or staff persons Abt or High/Scope who are 
usually more familiar with the testing procedures and expected learning out- 
comes than with other program aspects such as comprehensive services, federal 
expenditures, generation of in-kind commmunity services.) 



At the two Home Start summative sites without a He.id Start program, 
i , e , Kansas and OhiOr the major site visitor will spend three rather than four 
days. The major visitor and the monitor can give one day each to the gather- 
ing of Home Start program implementative data and two days to collecting 
Home Start cost information (both federal expenditures and in-kind.) 

Site Visit Data Specification - Fall *73 

1) Six Home Start Sites 

• Program Up-'Date and Future Plans 

During the Spring 73 visits to each of the 16 Home Start 
programs the locale staff cc«npleted a set of objectives for 
overall program activities and for each component. E^pbasis 
was placed on definite implementation events of the following 
six months which a future visitor could most likely verify as 
having occurred or not occurred* Approximate dates of planned 
occurrance and identification of the key staff person responsible 
for occurance were also identified and recorded on the "Future 
Plans" calendar. During FaJ.1' 73 visits of these implementation 
objectives will be reviewed. Verification of successful imple- 
mentation will be noted along with specific implementation proi)lems 
which prevented satisfactory completion of major events. Aides 
to successful implementation can also be noted* 

Major organizational changes will also be noted during this visit. 
This includes staffing changes, alteration of staff training and 
supervision procedures/ and the present role of parent involvementv 

» More Comprehensive Documentation of Hiring , Training , and Super- 
visision Practices for Hc^e Visitors 

The Spring *73 visits included in-the-home observation of four 
home visits per site. Local staffs were also interviewed regarding 
^idealized" and actual hiring, training, and supervision practices^^^^ 
This Fall the summative testing team will also conduct random obser- 
va tions of Home Visits randomly selected * (With each summative site 




tester completing '2 Homo Visit observations, eight to ten 
observations can be conducted of this plan--summative field 
operations--for use of a Home Visit observation checklist.) 

While this data will give us a description of the central 
Home Start treatment i.e , r the Home Visit, the comprehensive 
documentation of hiring, training^ and supervision is necessary 
to identify across program of strength and weaknesses in the 
installation or implementation of this treatment. Recommendation 
regarding continued improvement in the delivery of effective Home 
Visits can then be made, {Summative data may also be analyzed in 
terms of this central program process the hiring ^ training i and 
supervision of Home Visitors.) - 

# Verification of Projected Expenditure Patterns and Generation of 
In-Kind Community Services 

The Spring '73 visits gathered extensive expenditure and in^kind 
information. Interim Report III (August 30, 1973) will 
analyze this information* The Fall visits will determine whether 
actual expenditure and in-kind generation are consistent with the 
projections based on the Spring data. Further verification of the 
actual delivery of in-kind community services will be emphasized • 

2) Four Head Start Sites « 

# Documentation of Present and Projected Expenditure Patterns 
and of Generation of In-^Kind Community Services 

During approximately two days with each of these programs the 
major site visit person frcrfti the Abt or High/Scope staff will 
become acquainted with the Head Start Administrative staffs and 
collect baseline information such as present budgets and expendi- 
: ture reports, and present records on the securing of in^kind 

community services* Additional information regarding projection 
of future expenditures and in-kind generation will also be gathered 
through interviews with key staff. When feasible, large in-kind 
categories will be verified on-site with the contributing person 
or agency. 



# Overall Program Information on the Service Population 



While the summative staff will gather a variety of demographic 
data on families tested, the site visitors will secure information 
on the size and general characteristics of the population served. 
This data is necessary to compute per unit costs and to assist in 
the assessment of the degree of similarity of Hc^e Start and Head 
Start programs. 

Site Visits Procedures 

Program implementation data will be gathered through interview forms and 
^ checklists like those used in the Spring '73 visits. IJost data collection 

will rely heavily on actual program budgets and expenditure reports. In-kind 
services will be documented on a form (similar to the one used this Spring) ""^^^^^^^^^^^^^^^^^^^^^^ 
which number of units of service basis and on an estimate — and later verifiea-^ 
of the local cost per unit of service. 

Training for Site Visitors I . 

The major site visitors (3-4 days at each locale) will be persons who have 
been members of the National Evaluation Study Team at either Abt Associates 
or High/Scope, All will have participated in the training and conducting of ^ 
the Spring '73 site visits. As two of the major site visitors will visit two 
locales each/ only four major site visitors need be prepared for field visita-^ 
tion. A two day training session (at Abt Associates) is planned for late 
September. The three or four monitors gathering one day of program implemen- 
tation data will be trained in a one day session run concurrently with the 
September training of major site visitors. 

Training methods will include role playing, and protocols based upon the 
the actual instruments to be used. 

Analysis and Report of the fall J 7 3 Site Information y ' : - 



The report materials based upon the Fall site data will focus on the two 
central (nechanism of the Home Start treatment the activity of the Home 
Visitor in the Home, and the amount and kinds of in-kind ccMwnunity services 
which have been secured for Home Start families through the efforts of the 
Home Start programs* While the Home Visitor section of the report will des- 
cribe the content and types of in-the-house interaction, (and the recruitments 
training, and supervision of Home Visitors the other major delivery mechanism 
—comprehensive services— will be mostly described in terms of the ma gnitude 
of such services, i.e. , analysis of program expenditures will determine the 
amount and kind of comprehensive services bought for the families with program 
dollars. Analysis of in-kind services will allow the determination of the 
magnitude (or market value) of services which have been secured through pro- 
gram efforts. 

This report will also present initial descriptive Information on the projected 
expenditures and in-kind generation of the four Head Start programs visited. 
As much of this data will not be verified unit! the Spring '74 visits only 
very tentitlve comparisons will be made at this time of Home Start and Head 
Start cost data. 

Descriptive analysis ^of Home Start and Head Start cost data will include the 
following? cash costs, ^.evered costs (donated, iri-kind by type of source r) 
full costs (cash costs plus levered costs,) unit c;^sts (per family, focal 
parent, focal child, target child, family member,) and functional c6st$ 
(administration, career development, home visits (by component) parent 
develoxxnent, and occupancy. 

As the Home Start and Head Start cost data is presented and compared # the 
report must be prepared to make at least tentative statement regarding the 
degree to which these six Home Start programs and four Head Start programs 
are representative of the totality of such programs across the nation. This 
report will determine the representativeness of the six Home Start sites by 
examining implementation of key treatment variables within each of the 16 
Homn Start sites. Information on these variables was collected during [Spritt^ 
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while similar information may not be on treatment and output within the 
1000-2000 Head Start programs, the overall program characteristics of the 
four programs (size, budget i basic delivery modOi types of families served) 
can be compared to the universe of Head Start programs. The nature and 
size of the subset of Head Start programs represented by these four programs 
could be estimated* 

Projected Site Visits for Spring 74 

Spring *74 visits to the six summative Home Start sites and four Head 

Start sites would be similar in size and content to the fall 73 visits 

the collection and verification of cost data (expenditures and in-kind) v 

would be emphasized. 

All 16 Home Start programs would be observed from a program implementation 
point of view/ Previous objectives would be reviewed* Aids and deterrents 
to successful implementation would be noted. Changes in overall program 
organization would be recorded. At the ten non-summative sites a one day 
verification of projected expenditures and in-kind services would also be 
conducted. (Baseline data to be verified has been collected during the 
Spring '73 visits, ) 

In addition; the 16 Horne Start progr^mft will all be included in an across 
program cut at describing one key aspect of program implementation. During 
this and remaining site visits (fall 74 etc) a series of discrete cuts could 
be taken. Comprehensive checklists or inventories could be used to document 
such treatment variables as staff ing patterns and the comparing of community 
services^ the changing role(s) of parents in Home Start, staff or family 
stability. Such highly specified probes could produce a sharper picture of the 
key implementation issues faced by Home Start during this three year period, 
More highly defined recommendations for improvement might also be made* 



Chcscking the quarterly program reports for accuracy and com- 
pleteness. If clarifications are needed on the reports^ the 
programs will be contacted by Abt Associates via telephone. 

After the reports have been checked^ a Year Tally will be com- 
pleted for each individual program. Data is recorded on these 
tallies quarterly on Family Characteristics, Staff Characteris- 
tics! Referrals, and Financial Expenditures. 

The data on Family Characteristics, Staff Characteristics, 
and Referrals is then used to complete the Quarterly Pro- 
file Reports, reflecting totals for each Howe Start Program 
and the total for the National Home Start Program. 

Quarterly Profile Report totals are then recorded on the 
Three-Year National Profile. 

When the National Profile Reports have been compiled, copies of the Re*- 
ports will be submitted to the Office of Child Development. Copies of thq 
Year Tally Reports for each individual program will be sent to the prograias, 
together with comments regarding reporting procedures and Quarterly Profile 
Reports. 

An analysis oj: consolidated data will be computed quarterly. Scmt-anually, 
the National Profile Reprts will then ;:orm part of the data base for the case 
study and summary reports. 



